[PHP] Using cURL to download large XML files

Posted by ndg on Stack Overflow See other posts from Stack Overflow or by ndg
Published on 2010-02-20T10:54:29Z Indexed on 2010/03/14 20:05 UTC
Read the original article Hit count: 248

Filed under:

curl

|

php

|

transfer

|

large-files

I'm working with PHP and need to parse a number of fairly large XML files (50-75MB uncompressed). The issue, however, is that these XML files are stored remotely and will need to be downloaded before I can parse them.

Having thought about the issue, I think using a system() call in PHP in order to initiate a cURL transfer is probably the best way to avoid timeouts and PHP memory limits.

Has anyone done anything like this before? Specifically, what should I pass to cURL to download the remote file and ensure it's saved to a local folder of my choice?

© Stack Overflow or respective owner

Related posts about curl

iPhone Curl Left and Curl Right transitions

as seen on Stack Overflow - Search for 'Stack Overflow'
I am looking for a way to do a UIViewAnimationTransitionCurlUp or UIViewAnimationTransitionCurlDown transition on the iPhone but instead of top to bottom, do it from the left to right (or top/bottom in landscape mode). I've seen this asked aroud the internet a few times but none sems to get an answer… >>> More
PHP Curl and Curl

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi , I am able to send a get request using PHP Curl . But the same thing when i try from command line in Linux (/usr/bin/curl ) I am unable to do so. Please find below my PHP curl that is working $url = "http://172.20.22.26"; $headers = array("Host: 172.20.22.26", "User-Agent:… >>> More
php, curl , php curl , multipart/form-data , upload picture redirect

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm trying to upload some pictures using php cURL on a classified ad website .I think that I set all the parameters properly but I see that there is a kind of redirect after I post the picture . The issue is that the url where I'm getting redirected gives 404 error instead to return the html that… >>> More
Allow Incoming Responses from Curl On Ubuntu 11.10 - Curl

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I'm trying to get a Curl Response from an outside server, however I noticed I cant neither PING the server in question nor connect to it. I tried disabling the iptables firewall but I had no success. My server is running behind a Cisco Linksys WRTN310N Router with the DD-wrt firmware Installed. In… >>> More
cURL works but PHP cURL fails to internet [migrated]

as seen on Pro Webmasters - Search for 'Pro Webmasters'
Trying to diagnose an issue using PHP to cURL to an Internet location on a RedHat Linux server. cURL is installed and working, and: <?php var_dump(curl_version()); ?> shows all the correct information in the output. The issue is I can use PHP to cURL to localhost on the box itself, but… >>> More

Related posts about php

Magento, NGINX, PHP-FPM, APC, MEMCACHED, 16gb Ram CentOS, Spiking PHP-FPM to 100% CPU

as seen on Server Fault - Search for 'Server Fault'
I have been trying to resolve my issue of spiking cpu caused by php-fpm processes. I've reduced the php-fpm config settings to: pm = ondemand pm.max_children = 12 pm.start_servers = 2 pm.min_spare_servers = 2 pm.max_spare_servers = 10 pm.max_requests = 500 php_admin_value[memory_limit] = 128M Problem… >>> More
PHP Pear Installation on CentOS

as seen on Server Fault - Search for 'Server Fault'
[root@ip ~]# yum install php-pear* Reducing CentOS-5 Testing to included packages only Finished Setting up Install Process Package 1:php-pear-1.8.1-2.el5.centos.noarch already installed and latest versio … >>> More
Apache configurations for php "AddType text/html php" or "AddType application/x-httpd-php php .php"

as seen on Server Fault - Search for 'Server Fault'
I am taking over an application server and discover that it contain the following settings: AddType text/html php Although it works, but my understanding is that it should set as following: AddType application/x-httpd-php php .php What are the key differences between the two settings?… >>> More
mod_rewrite settings causes server to throw HTTP 500 errors instead of 404

as seen on Server Fault - Search for 'Server Fault'
Hello. I have a server with VBulletin forum (working under Apache 2.2, CentOS). The default settings for it in .htaccess are as follows: RewriteEngine on RewriteCond %{HTTP_HOST} ^gsmforum\.ru RewriteRule (.*) http://www.gsmforum.ru/$1 [R=301,L] # If you are having problems or are using VirtualDocumentRoot… >>> More
Problems installing Memcache (PECL extension)

as seen on Server Fault - Search for 'Server Fault'
I have installed memcached fine, and now I will need to install PECL extension memcache. Im running RedHat x86_64 es5. The installation gives me this: downloading memcache-2.2.6.tgz ... Starting to download memcache-2.2.6.tgz (35,957 bytes) ..........done: 35,957 bytes 11 source files, building running:… >>> More